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ABSTRACT 



This paper examines new approaches to program evaluation and 
explores their possible utility in Latin American educational settings. Part 
1 briefly discusses why new ideas for evaluating educational studies are 
needed. Part 2 examines seven new evaluative approaches as follows: (1) 

"Concept Mapping, " a type of structural conceptualization; (2) "Participatory 
Self-Evaluation, " designed by United Nations development programs; (3) 

"Social Cartography, " an interpretive tool based on textual analysis and 
spatial patterning; (4) " Intertextual Evaluation, " based on 

post-structuralist literary theory, establishing useful criteria for program 
evaluation; (5) "The Analogy of the Soil Scientist," a sociocultural approach 
for program planning and evaluation; (6) "Evaluation and Organizational 
Learning, " a cybernetic conception including dialogue, experimentation, and 
team work; and (7) "Organizational Perspectives, " an evaluative approach 
employing five perspectives : managerial hierarchy, street -level bureaucracy, 
organizational development, conflict and bargaining, and chance and chaos. 
Part 3 compares and contrasts the seven approaches and discusses their 
applications to Latin American programs. (Contains 10 figures and 44 
references . ) (RIB) 
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We need to think of evaluation as an “eclectic enterprise,” 
with a rich tool kit of methods and many legitimate 
perspectives, purposes, questions, and uses. 

(Chelimsky 1997, 25) 

A discussion is just beginning in educational settings, addressing the rights of the various 
social subjects to participate in decision making and in the production, development, and 
evaluation of the process of teaching and learning — a discussion that will likely remain 

central in the region’s pedagogical and political thought. 

(Torres and Puiggrds 1995, 27) 

Changing conditions at global and local levels in the production of knowledge as 
well as the introduction of new educational models in Latin America, justify the search 
for new ideas in the field of program evaluation. Accordingly, this article examines seven 
new Anglo-American evaluation approaches, maps them as an intertextual field, and 
explores their possible utility for Latin American educational settings. The perspectives 
examined are concept mapping, participatory self-evaluation, social cartography, 
intertextual evaluation, the analogy of the soil scientist, organizational learning, and 
evaluation based on organizational analysis. These approaches, while they do not attempt 
to provide comprehensive models applicable to all kinds of situations and contexts, seem 
to offer the possibility of more participatory and constructivist practices of evaluation in 
education and related fields. 

Evaluation has not traditionally been a favored activity in Latin American 
education. When it has taken place, attention has been paid mainly to quantitative factors, 
without promoting the participation of beneficiaries or ground-level agents. Several 
authors have pointed out that the characteristics of the social and educational systems of 
the region would require a different approach. The deep cultural differences, the necessity 
for global and decentralized diagnosis, the possibility of strengthening the activities of 



the grassroots communities, and the organizational characteristics of schools are some of 
the inherent elements of the region from which we reflect about the possible utility of the 
seven perspectives mentioned above, taking into account that the applicability of these 
perspectives to the Latin American context can only be fully established in the evaluative 
praxis. 

The evaluation of projects and programs is a relatively young discipline that 
seems to be gaining importance around the world. Its development has required the 
incorporation of theoretical and methodological elements from a variety of areas of social 
and human research (Chelimsky 1997). As Greene (1994) states, “program evaluation is 
integrally intertwined with political decision making about societal priorities, resource 
allocation, and power” (531). We believe that reflection about some new methods of 
evaluation and their potential utilization in the region will contribute to the possible 
improvement of educational systems, both in terms of efficiency and of equity and 
quality. 

Our analysis does not include all of the new ideas that theoretical 
conceptualization and the practice of evaluation have generated in the past few years. Nor 
are we arguing that the perspectives we present in this paper are necessarily the best new 
approaches in the field. Rather, our purpose has been to look at some approaches as a 
flexible set of evaluation tools that might be adapted to different socio-cultural 
conditions. 

In the first part of this study, we briefly justify the search for new ideas in the 
evaluation of educational programs and policies. The second section is a characterization 
(and mapping) of seven ideas or approaches recently developed in this field. Then we 
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examine some characteristics of the Latin American context, and how the new 



perspectives identified could be useful for evaluation practice in the region. 



Why do we need new ideas in the evaluation of educational programs in Latin 
America? 

During the last several decades, the grand social and educational theories created 
in the time of modernity have been at least challenged and brought down from their 
pedestals, at the same time that the patterns of political and social interaction have 
suffered important modifications. With the irruption of post-modernity, basic notions 
about reality and the production of knowledge have experienced a process of decentering 
and fragmentation. 

Guided by the purpose of responding to these changes, Lincoln (1994) notices the 
need of making evaluation more flexible, more adaptable to the changing character of 
different contexts and the need to incorporate new actors into the evaluation process, 
especially the need to recognize and include many voices and perspectives. In a similar 
vein, Chelimsky (1994, 1997) points out that one of the main characteristics of 
evaluation, as it is being practiced today, is the presence of multi-methodological designs 
that use the advantages of one method to complement or to make up for the deficiencies 
of other methods. In this way, it is possible to combine a variety of quantitative and 
qualitative methods. Many of these changes result in evaluators who present themselves 
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more as translators and collaborators, rather than as judges of a particular program or 
project. These kind of approaches can be contrasted to the more traditional perspectives 
in the field. The work of Rossi and Freeman (1985), for example, can be considered as a 
traditional approach in program evaluation. These authors focus on the use of 
experiments and cost-benefit analysis, where stakeholders have a very passive role. 

The search for new theoretical paradigms and more flexible practices that attempt 
to leave aside the existing paradigms is a reflection of political, economical and social 
changes that affect the entire world (i.e., the end of the cold war, the fall of communism, 
the information economy, etc.). The deep social changes that have shaken our world 
obviously include the educational arena. Education in its variety of forms is very different 
today from twenty or even ten years ago. These changes also affect education in Latin 
American countries. 

Trying to characterize the status of education in Latin America, Torres and 
Puiggros (1995) talk about an organic crisis that affects school practices. Schools have 
become highly permeable to external influences and, to a great extent, the learning 
process has moved to other settings. The influence of the media and the disruption of the 
codes of communication between parents/teachers and pupils place new challenges into 
the educational process. Tedesco (1994) holds that there is a new model of relationships 
between education and society in Latin America. Institutional reforms linked to new 
models of decentralization that seek to improve quality and efficiency are at the core of 
those changes, together with the awareness that it is necessary to give a privileged place 
to education in national and local political agendas. 
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The evaluation of educational programs and policies is one element, among 
others, that may help to face this situation of crisis and change in different ways. As we 
show in the paper, some new approaches might contribute to a more participatory and 
constructivist practice of evaluation. As Torres and Puiggros (1995) point out, new social 
subjects are demanding to be part of educational decision-making processes in the region. 
The exploration of new evaluation approaches might well provide some clues on how 
students, teachers, parents and school administrators could become more important 
actors. 



Seven new approaches 

In this section we describe seven ideas or approaches for the implementation of 
program evaluation, which have recently been developed in the Anglo-American context. 
The following analysis presents a brief characterization of each approach and a 
preliminary assessment of strengths and weaknesses. We recognize that the strengths and 
weaknesses might be better appreciated if each approach were related to an specific 
evaluation context, and that the weaknesses that we point out here could be, in many 
cases, overcome with a combination of techniques and approaches. 

First of all we analyze a method of structured conceptualization —concept 
mapping— that seeks to provide a graphic representation of the ideas or perceptions of 
program participants about a particular aspect of a given program. The second approach 
under consideration is the participatory self-evaluation method of Uphoff (1991), in 




6 



7 



which groups of beneficiaries of the program are invited to evaluate their own 
performance. Then, we describe social cartography, an interpretative method that has 
been used in the field of comparative education to map debates and to situate and inter- 
relate different perspectives. The fourth idea to consider, intertextual evaluation, is an 
application of concepts of post-structuralist literary theory to the field of evaluation. 

The last three approaches are more general and tend to combine quantitative and 
qualitative techniques. The analogy of the soil scientist, for example, is a socio-cultural 
approach that seeks to provide a conceptual framework for the evaluation of social 
development programs. The two remaining approaches are based on organizational 
analysis. One draws upon an association between evaluation and organizational 
development and the notion that organizations can become structures open to constant 
learning. The other provides different perspectives that are combined to get an evaluation 
that accounts for the various organizational dimensions that affect a specific program. 



(1) Concept Mapping (Trochim 1989) 

Perhaps the least new of the approaches to be analyzed, concept mapping has 
been developed as a type of structured conceptualization by William Trochim (1989). 
The main objective of this method is to generate a conceptual framework as a guide for 
the planning and evaluation of programs and projects. It claims to be based on the ideas 
and/or perceptions of the participants of the program, which are represented in graphic 
form in a drawing or map. Applied to evaluation, concept mapping can be used at the 
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design stage, i.e. the identification of questions to be answered, topics of interest, node of 
evaluation, or as a technique of conceptualization of the ideas or perceptions of a specific 
group, i.e. beneficiaries, program administrators, etc., during the data collection stage. It 
is important to keep in mind that the concept maps are heuristic tools and do not claim to 
provide or model an objective description of reality. 

This method has several steps. The first establishes who is going to participate 
and what is going to be the focus of the conceptualization. Trochim states that there 
should be a broad representation of all the groups involved in the program, but 
participation can be limited to specific actors. Once the focus of the conceptualization is 
defined, the generation of ideas is generally achieved by the free expression of statements 
about the topic ("brainstorming"). A third step is the structuration of the ideas that have 
been expressed. The group assigns a numeric value to each idea that has been expressed 
according to the importance in relation to the focus of the evaluation and each individual 
organize the ideas in different groups according to his/her personal criteria. Then the 
individual results are combined. The fourth step involves the visual representation of the 
group’s perceptions. The representation implies a set of mathematical analysis that allow 
the placement of each idea in a space; to associate ideas in groups according to their 
conceptual proximity; and to establish a numerical value to each group of ideas (see 
Figure 1). The participants give a name to each group of ideas, discuss to what extent the 
resulting graphic representation is accurate, and make consensual modifications to the 
map. In the final step, the group has to decide what is going to be the use of the map 
according to the focus previously defined (e.g.: evaluation of results, problems of 
implementation, modifications to the program, etc.). 
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One of the strengths of this method is that it generates a visual representation of 
the main ideas and perceptions of the group. At the same time, it allows a high degree of 
participation in a process that takes relatively little time. In addition, this method has been 
combined with qualitative interviews with successful results (see Wiener et al. 1994). 

A weakness of concept mapping is that it assumes the existence of consensus 
within the participant group. It assumes that the members can arrive to an agreement 
about the final representation, which will not happen in the case of strong discrepancies 
and disagreements in relation to objectives or priorities or in relation to the interpretation 
about how is the program working. The method also has a limited utility since it provides 
only a description of the perceptions of the group, and not a deeper comprehension about 
problems in the program. 



(2) Participatory Self-evaluation (Uphoff 1991; Mausolff 1996) 

Norman Uphoff has designed this method for development programs of the 
United Nations as a complementary technique to the traditional methods of evaluation. It 
advocates that the participants evaluate their own performance with the aim of increasing 
the ability of groups to satisfy their needs and to help achieve the objectives of the 
program. The method consists of group analysis of questions related to the program and 
the performance of the group. Each question is seen to measure an aspect of the 
performance of the group, and the group has to decide which value to assign to each 
question. 
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Uphoff suggests that groups should decide between four alternatives for each 
question: the first alternative indicates an optimum performance in the indicator or 
indicators selected to measure the item (3 points); the second one, a satisfactory situation 
in which it is possible to make some improvements (2 points); the third one, a non- 
satisfactory situation in which it is possible to make substantial improvements (1 point); 
the fourth one reflects the worst possible situation (0 points). The overall performance of 
the group can be determined by an average of the total of the points obtained. 

Uphoff points out that the score that each group obtains is not as important as the 
interchange of ideas and the reflection that is generated in the group discussion as well as 
in the decisions that the group makes for the improvement of those aspects that are 
perceived as deficient. From the point of view of the program administrators, the method 
might be used to identify how participants see strong and weak points and to evaluate the 
progress of each group by comparing current assessments with previous self-evaluations. 

Uphoff recommends that, as a first step, the administrators of the program discuss 
the purpose of the evaluation and the utility of a participatory self-evaluation. Then, they 
can prepare a tentative list of questions and indicators to measure the group performance 
in each item. This allows for the method to be discussed with the key participants or 
those beneficiaries most involved with the program. It also allows for changes in the 
formulation of questions, the inclusion of new questions, and the elimination of others. 
When put into practice, a member of the group chosen for that purpose coordinates the 
process of examining each question and establishing which is the answer that best reflects 
the ideas of the members about their performance, registering the scores obtained in each 
question or item. Throughout the process, members of the group have the freedom to 
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include new items and dismiss others, according to what they consider their real 
objectives. 

Besides Uphoffs practice of participatory self-evaluation for the United Nations, 
Mausolff (1996) used this method with seven rural communities in Honduras. The author, 
interested in applying an approach that expressed the cultural diversity at the local level, 
sought to measure the perception of each community in relation to the group objectives 
and the objectives of the project with which they were associated. Different from the 
scheme proposed by Uphoff, the groups of Honduran peasants developed the questions 
starting from the identification of objectives in a more independent way, without the 
participation of program administrators or directors. From the self-evaluation of the 
communities, Mausolff designed a “social map” of the participant groups according to 
their perceptions of privileged objectives (see Figure 2). Based on his fieldwork, 
Mausolff concludes that “participatory self-evaluation provides capacity building by 
directly involving project beneficiaries in the evaluation process” (p. 284). 

It appears to us that a main benefit of applying the method developed by Uphoff 
is that it would allow for a more active involvement of participants in the search for 
solutions to problems or deficiencies that they recognize. In this sense, the effectiveness 
of the measures geared toward the improvement of some aspects of the program might be 
greater than in the cases where the problems and solutions are identified by an external 
agent or imposed by the program administrators. Another positive element is that, 
assuming the objectives of the program are realistic, the search for solutions or strategies 
of improvement takes place by focusing on the resources of the group itself rather than on 
what can be provided from outside the group. In this way, self-evaluation can contribute, 
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Figure 2. “The social geography of seven rural Honduran community groups: Peasant organization 
involvement” in Mausolff (1996). 

This map— based on the results of the auto-evaluation— shows how the groups can be patterned in 
relation to their views of four kind of objectives. 



as Uphoff points out, to increase the capacity for negotiation and improvement of group 
interactions. Nevertheless, the method seems difficult to apply when the program is not 
oriented towards beneficiary groups but towards organizations and individual persons. It 
also requires a high degree of commitment by the participants to the group activity and to 
the program or project of which they are part. 



3) Social Cartography (Paulston 1993; Leibman and Paulston 1994) 

Social cartography is based on textual analysis and spatial patterning; it is 
interpretive and its purpose is to situate ways of seeing phenomena in relation to one 
another in a heuristic visual bi-dimentional representation. These spatial constructs re- 
inscribe and interrelate diverse -and frequently incompatable— perspectives on social and 
educational phenomena, ways of seeing which can be present at an intertextual or a 
practical level. The approach seeks to respond to the post-modern condition of 
uncertainty following the decline of the main social discourses of modernity (e.g.: 
Marxism, positivism, etc.). Accordingly, it intends to account for and reactivate most of 
the possible perspectives, avoiding a hierarchical order and considering any textual 
representation as a product subject to constant revision, questioning, and mapping. 

Leibman and Paulston (1994) identify three types of metaphorical maps. 
Phenomenographic maps present information that result from the investigation of 
different theoretical perspectives -different conceptualizations-, forming a “cartography 
of thought” (Figure 3). Conceptual maps represent relations between different 
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Figure 3. A phenomenographic/conceptual landscape of contemporary theories in education 
discourse. Source: Paulston (1994). 
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phenomena, more open to the ideas and visions of the cartographer (Figure 4). The third 
type are the ones called mimetic maps, which account for a deconstructivist mimesis, 
representing the visual juxtaposition of phenomena or objects according to a specific 
cultural or theoretical perspective (Figure 5). 

Applied to evaluation, social cartography can be used with different purposes: the 
cartography of previous perspectives on and evaluations of the policy or program; the 
cartography of other programs or policies previously or simultaneously developed with 
regard to the same topic, establishing a relationship to the program or policy evaluated; 
the cartography of the interpretations or ideas of the groups involved in the program 
through discourse analysis, which can be done through the interviews or written 
documents. The implications of applying social cartography to program evaluation can be 
appreciated taking as reference the work of Guba and Lincoln (1989). These authors have 
developed an evaluation model that intends to include in an equal bases all of the people 
involved (stakeholders) and their constructions or visions of the program. Here, the task 
of the evaluator is to carry out a process of negotiation between all the different 
stakeholders and their “constructions.” This process, we are told, enriches all of the 
participants and, ultimately, generates a consensus, or a final construction agreed upon by 
all stakeholders (see Figure 6). Our opinion is that, even though it is healthy to include all 
of the voices and to enhance discussion among stakeholders, consensus is not always 
achievable. Social cartography, in contrast, allows the presentation of diverse existing 
“constructions” as a field of difference, without pretending to arrive to a consensus 
among all the people involved in the program, as it is shown in the example of the 
evaluation of a program of university reform in Nicaragua (see Figure 7). On the other 
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Figure 4. A conceptual landscape of contemporary ideologies. Source: Graham (1992). 
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Figure 5. A mimetic map exhibiting phenomenographic and conceptual styles. Source- Eaton in 
Gould and White (1986). 
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Figure 6. “The Methodology of Constructivist Inquiry” in Guba and Lincoln (1989). 

The authors propose a rational and logocentric model in which the outcome of the evaluation is a 
consensual construction. 
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Figure 7. “A micro mapping of educational and social change theories in Nicaraguan higher 
education reform practice” in Paulston and Ripperberg (1991). 

This map shows how the social cartography approach can present the different theoretical 
conceptions affecting a program. See that the theories are presented in a situated and ironic way, 
and that the mapper is located in the figure as well. 



hand, the social cartographer accepts that each map is a partial and provisional 
interpretation. While Guba and Lincoln’s model acknowledges that evaluation should be 
conceived as a process open to continuous enrichment and new interpretations, it does not 
seem to recognize that its final result is no more than a “construction” of the evaluator, or 
a position in the intertextual debate. 

Social cartography, when it is applied to evaluation through the analysis of the 
perspectives of the participants, has common points with the concept maps of Trochim 
(1989). However, their methodologies are very different since the approach developed by 
Trochim assumes an essentialist reality and tries to mirror or reflect, by mathematical 
methods, the ideas of the participants. Social cartography, in contrast, proposes an 
interpretation of how different discourses interact or how to visualize spatial relations of 
the program. While the first one seeks a common vision, the second one assumes the 
existence of diverse perspectives in constant intersubjective patterning. 

The strong points of social cartography are that it allows us to visualize -and 
eventually to heuristically integrate— different perspectives that conceptualize the 
phenomena, and how they interact or interrelate (as shown in Figures 3 and 7). Therefore 
it can constitute a good source of relational knowledge about how the program is seen to 
exist. In addition, it appears as a valid attempt to respond to the challenge of post- 
modernity as it opens space for radically different perspectives of knowledge and reality 
without assuming a dominant ontological position or central tendency. 

One of the limitations of this approach is that it reflects the interpretation of the 
evaluator or cartographer, which can be very different from that of the administrators or 
the program funders. On the other hand, the intertextual map, which intends to appear as 
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a possible interpretation among many others and open to constant revision, might all too 
easily come to be seen-and misconstrued-as a true “model” of reality (Nicholson- 
Goodman and Paulston 1996). 



4) Intertextual Evaluation (Roe 1992) 

Intertextual evaluation is based on the post-structuralist literary theory of Michael 
Riffaterre and shares the indeterminate world of social cartography 1 . Here the text is also 
interpreted and “evaluated” in relation to other texts. Its objective is to establish which 
criterion should be used to evaluate a program or policy, particularly to find an intertext 
or metacriterion that accepts the conflict or proposed polarization found in diverse textual 
accounts. 

This approach is applicable when there is conflict over how a policy or program 
should be judged, i.e., mainly according to the objectives determined prior to its 
implementation, or according to the impact that it had in the way that it was 
implemented, or according to other possible criteria that can be mutually exclusive 
(efficiency, equity, cost-efficiency, etc.). Roe provides an analysis of how the policy or 
program to be evaluated sets, in its implementation, the criteria to be used, resulting in 
the common formula of “each case should be evaluated according to its own merits.” The 
approach is also applicable to the programs or policies that generate polarization between 
two opposite positions and here the objective is to determine a metacriterion that accepts 
the polarization and allows the generation of recommendations about the path to follow. 

ERIC 




Taking an example developed by Roe (1992), the evaluation of a policy 
established by an university in order to respond to the request of Native Americans for 
the devolution of the remains of their ancestors in the hands of museums, the method 
implies the identification of the “idiolect,” in this case the demands of the native groups 
requesting that the returning of the remains, in opposition to the “sociolect,” the 
traditional position of the museums and the scientific community in favor of keeping 
those remains for their study. The second step consists of identifying the 
“ingrammaticalities” of the discourse that reflect the tension or existing contradiction, 
which allows in turn the identification of an “intertext.” In this case, the insistence in 
terms like communication, participation, consultation, and dialogue -the 
“ingrammaticalities”— are conducive to identify the “intertext” as the fact that those who 
should decide about the topic are the natives whose remains are in debate. Roe considers 
that this element provides a metacriterion that includes and sustains the opposition 
“sociolect/idiolect”, and that it is conducive to consider the criterion of evaluating each 
case (the remains demanded by natives) in particular. 

One of the strengths of intertextual evaluation is that it accepts conflict and seeks 
a metacriterion that allows resolving the polarization avoiding the preponderance of one 
position above the other. Its weaknesses are related to the difficulty of applying it when — 
even in the cases of controversial topics— the funder or the administrator of the program 
predetermines the criterion. On the other hand, this method depends on documents and 
assumes that, with exegesis —or close reading— reality can be found in the text. Like 
social cartography, it is based on the interpretation of discourse or texts; however, social 
cartography does not attempt to find the resolution of the conflict within the discourse 
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and does not establish a linear scheme as is done by the intertextual evaluation proposed 
by Roe [(idiolect = sociolect) => intertext)]. It is also a difficult method to be explained 
and communicated to participants of a program. It is, perhaps, more applicable to the 
analysis of policies than to the evaluation of social projects or programs. 



5) The analogy of the soil scientist (Klitgaard 1995) 



This is a socio-cultural approach for development program planning and 
evaluation. Its starting point is that the project design and techniques should be able to 
adapt to a variety of different socio-cultural contexts. Klitgaard points out that those 
methodologies that previously have attempted to take into account diverse contexts have 
encounter severe difficulties in establishing the type and degree of interactions that are 
produced between the implemented programs and the socio-cultural conditions. It does 
not appear to be possible to construct models that predict this type of interaction. 
Traditional evaluation methodologies, in general, have not taken into consideration the 
fact that socio-cultural conditions present a changing reality in constant transformation 
that is highly variable even within the same geographical region. 

The analogy that Klitgaard proposes as one way out to this situation is based on 
the idea taken from Putnam (1993) that socio-cultural conditions are the symbolic ground 
in which the development process takes place. If, as he argues, the conditions of that 
symbolic ground determine, to great extent, the viability of a project, then the evaluator 
should behave as an soil scientist: 

2 7 




17 



"To carry out their practical work, soil scientist analyze soils, using partial and 
incomplete measurements of soil differences. Their typologies and empirical 
results do not pretend to capture all factors characterizing a soil area. Good soil 
scientists listen carefully to what local farmers say about their land and their 
farming practices. Beyond just describing differences in soil conditions, they 
study the interactions among soil types, crops, and soil treatments... Soil scientists 
often rely on experts from other disciplines --agronomists, chemists, economists, 
geographers— to help them assess local conditions" (Klitgaard 1995, 143-144) 



Klitgaard acknowledges that cultural change is a more problematic and 
controversial matter than the treatment of soils. The objective of applying this analogy to 
the work of the evaluator is to find partial but useful indicators of the socio-cultural 
conditions that may be affecting the program and that can influence -if not explain— its 
instrumentation and results. These indicators should emerge preferably from the 
knowledge of the dwellers of the area and the local experts in different disciplines. The 
evaluator should provide his or her theoretical and comparative perspective as a 
complementary element and should seek, as a final objective, to help the participants 
make decisions when they require assistance in changing socio-cultural conditions. 

We see the strong point of this approach in that it provides a wide perspective of 
the program and its connections with the socio-cultural context, therefore attempting to 
resolve some of the traditional limitations of the evaluations of development programs. 
As Bamberger (1991) points out, in developing countries the evaluation of projects is by 
and large financed by governments or international agencies and does not usually include 
the intended beneficiaries of the projects, nor respond to implementation problems. Even 
though the analogy of the soil scientist is explicitly directed to social development 
programs in the context of international assistance, it is not difficult to imagine its 
application in the educational context. 
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A weakness of the approach is that there are no examples of its application. As 
Klitgaard recognizes, in order to prove the utility of this type of evaluation it is necessary 
to advance in the gathering of “decentralized” socio-cultural information related to 
development indicators. Apparently, its application requires more resources than 
traditional approaches; therefore it is likely to find financial obstacles or obstacles related 
to the lack of time for implementation. This approach also depends, to a great extent, on 
the collaboration of participants and their ability to articulate the data required. 



6) Evaluation and organizational learning (Preskill 1994) 

We see this approach as originating in a cybernetic conception of organizations. It 
is a view that privileges dialogue and the discussion of ideas, experimentation with new 
procedures and practices, and team work (Morgan 1986). The possibility of linking 
evaluation to the concept of organizational learning has been explored by various authors 
(see Torres 1994; Leeuw et al. 1994). 

Preskill (1994) proposes that the evaluator should concentrate in the area of 
development of human resources as a first step to introduce the organizational learning 
approach. In this context, the evaluator assumes the role of "facilitator" of the process by 
which the organization sets in motion mechanisms geared toward establishing a culture of 
participation and discussion of ideas and the use of information to implement internal 
changes. This idea supposes the incorporation of evaluation to the dynamics of 
organizational change. The evaluator should help the members of the organization 
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become aware of the processes of learning that occur in the daily routine of work and to 
develop a positive attitude toward change. This top-down approach, then, allows a 
displacement of the traditional focus of the evaluation from particular programs to 
organizational development. 

According to the model proposed by Preskill, after a period of training in the 
practice of the main elements of this new organizational conception (team work, critical 
thinking, negotiation, self-evaluation, etc.), the members of the organization should take 
part in a participatory evaluation of an aspect or project of the organization (see figure 8). 
The use of three foci is of fundamental value to make evaluation contribute to implant a 
culture of organizational learning: (1) the reflection about the objectives, process and 
results of the evaluated project; (2) the resulting dialogue from the contact with other 
organizations involved in similar projects and the analysis of relevant bibliography; and 
(3) an action plan that translates what has been learned into organizational changes. 

The concept of organizational learning is reflected in the "school restructuring" 
movement (Lieberman 1995), recently in vogue in the USA and Canada, which promotes 
initiatives toward shared decision making among administrators, teachers, and parents, 
more collaboration among teachers, and experimentation of new pedagogical practices. 

The strong point of the method proposed by Preskill is, perhaps, that it recognizes 
the importance of organizational culture and facilitates the utilization of participatory 
techniques. On the other hand, its application depends on a collaborative and 
participatory attitude by the part of administrators and other members of the organization. 
It requires long time interactions between the evaluator and the organization, and it is 
difficult to apply by an external evaluator. Another weakness is that it ignores the politics 
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of organizational life, and the perspectives of beneficiaries or clients, a particularly 
important omission when the organization aims to render a service to an external 
population that might not share an assumed consensus of problems, treatments, and 
desirable outcomes. 



7) Organizational Perspectives (Rogers and Hough 1995) 

This approach seeks to evaluate a program from five different organizational 
perspectives: managerial hierarchy, street-level bureaucracy, organizational development, 
conflict and bargaining, and chance and chaos. Four of them have been taken from the 
models of program instrumentation suggested by Elmore (1978). Rogers & Hough (1995) 
add a fifth perspective -“chance and chaos” — underlying that each perspective provides 
particular and limited scenarios and approaches (see Figure 9). In order to be effective, 
they contend, program evaluation should consider all five perspectives. 

“Managerial hierarchy,” the first perspective, explicitly focuses the evaluation on 
the established objectives of the organization or program, and tends to take into account 
mainly the needs of the directors of the program. According to a second perspective, 
“street level bureaucracy,” the evaluation would seek to determine how the program 
actually functions by observing the interactions of the street-level agents and 
beneficiaries. A third perspective, “organizational development,” implies evaluations 
with high participation of the members of the organization in its design and 
instrumentation. “Conflict and bargaining” are the main elements of a fourth perspective. 
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which considers the different and sometimes contradictory objectives of the groups that 
influence program implementation. “Chance and chaos,” the last perspective to consider, 
originates in the idea that organizations are subject to unexpected and unpredictable 
changes that may significantly affect their operation; therefore evaluations should 
especially consider the context of the program and its capacity for adaptation to change. 

As is easily seen, the organizational development perspective - one of the five 
that Roger and Hough propose - has points in common with the approach of 
organizational learning advocated by Preskill. However, Rogers and Hough’s approach 
recognizes the limitations of this particular perspective and seeks to complement it with 
other dimensions of the organization, while the second one, organizational learning, is 
limited to evaluation carried out within organizations that privilege this particular 
dimension. 

The advantages of the multiple perspectives approach is that it takes into account 
many of the cultural and political factors of organizational life, while allowing a large 
vision of the program and its relation with different dimensions of an organization. 
Possible limitations may follow from the fact that the focus of the evaluation is geared 
toward a conceptualization of programs from the organization standpoint at the expense 
of the perspective of the beneficiaries or clients. It should also be noted that this approach 
would seem to require more time and resources than traditional approaches. 
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Considerations 



We have presented seven new approaches in the field of program evaluation and 
attempted to delineate their main features. Nine figures help to illustrate their utility and 
originality . As it has been pointed out in the description of each approach, these seven 
approaches share many common features. One common characteristic in almost all is the 
consideration of diverse -even incommensurate— perspectives. With the exception of the 
method of concept mapping, which assumes a high degree of consensus, the rest of the 
approaches allow for a multiplicity of perspectives. Participatory self-evaluation, for 
example, suggests that the perspective of the beneficiaries might be different from that of 
the directors and other agents of the program, and different as well from that of the 
external evaluator. Social cartography opens space to include the multiplicity and 
diversity of different views of reality in any social scenario where the grand narratives of 
modernity are not accepted as comprehensive and universal views, but as competing 
mininarratives or perspectives. Intertextual evaluation is based on a polarization of 
perspectives, which are present in an intertext or metacriterion that does not eliminate but 
includes all. The analogy of the soil scientist, in turn, seeks to integrate the perspective of 
the evaluator with those of the program beneficiaries and experts in several areas. The 
concept of organizational learning is based on the principles of adaptation to change and 
participation of all members in an open debate of ideas, which allows for the benefit of 
different perspectives, although it is not clear to what extent different interpretations 
would be tolerated if organizational effectiveness is threatened. The approach of 
organizational perspectives, finally, originates in the idea that an effective and useful 
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evaluation can be constructed only through the consideration of the different dimensions 
of perceived organizational reality. 

Another feature shared by most of the approaches is the inclusion of various 
stakeholders in different phases of the evaluation. Participatory self-evaluation is based 
on the same idea that the beneficiaries should have a prominent place in the evaluation. 
The method of concept mapping promotes a wide participation for the formulation of 
ideas and for the construction and interpretation of maps. At the same time, it also 
recommends that the participants decide the focus of the evaluation and its utilization. In 
the same way, the analogy of the soil scientist and the approach of organizational 
learning encourage the participation of actors who occupy the base of the organization 
and who have in past been ignored. The method of multiple organizational perspectives 
recognizes the need to promote the participation of different groups, groups that may 
influence the implementation of the program. 

Various combinations among these approaches are certainly possible. In the 
framework of evaluations guided by ideas of organizational learning, the analogy of the 
soil scientist, or the analysis of the different organizational perspectives, concept 
mapping as well as participatory self-evaluation could be applied to achieve the 
participation of beneficiaries or ground-level agents. Social cartography could be also 
integrated to such approaches in order to situate and inter-relate the different perspectives 
of those involved in the programs or organizations. The work of Maussolf (1996) already 
described is a good example of the combination of participatory self-evaluation with 
social cartography. Intertextual evaluation could be used to conceptualize conflicts that 
arise from development programs or conflicts that emerge from organizational dynamics, 



by its utilization in evaluations that respond to the analogy of the soil scientist, or to the 
approaches of multiple perspectives or organizational learning. We think that it is very 
important to consider the possibility of combinations and even to consider the seven new 
approaches as part of a “rich tool kit” for evaluators today (Chelimsky 1997, 25). 

The seven approaches have in common that they do not claim to provide all of the 
answers to the problems we face in doing evaluations, or to apply to all the possible 
settings for evaluations. They just try to offer a method for a particular aspect in some 
cases, or general guidelines for a particular kind of evaluation in other cases. Evaluators 
and stakeholders, however, should be aware of the different implications that these 
approaches have. Methodological and political implications can be suggested using two 
dimensions: the participatory versus expert-based dimension (considering the level and 
scope of stakeholders’ participation) and the constructivism versus ontological 
essentialism dimension (see Figure 10). These two dimensions appear as particularly 
relevant in our analysis of the seven methods, and have been recognized as important 
issues in the field (see Greene 1994; Guba and Lincoln 1989). On the other hand, we 
acknowledge that others may consider that there are more critical dimensions that should 
serve for mapping the field. The positions of the approaches in the map are also 
contestable (or a matter of the mappers’ perspective). In addition, the map does not 
present all of the approaches that have been developed in program evaluation. We do 
include Guba and Lincoln (1989), whose arguments were considered in the analysis of 
the method of social cartography, and Rossi and Freeman (1985) as an example of a more 
traditional positivist approach. 
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Figure 1 0. A map of approaches in program evaluation presented as a relational intertextual field. 



As we showed in the previous analysis of each approach and in our maps, these 
are seven new ideas that depart from the traditional positivist, expert-based evaluation 
approach that under modernity prevailed in Latin America and around the world. We are 
aware that naturalistic, constructivist and participatory methods have been developed at 
least since the 1970s. We consider, however, that a positivist (or post-positivist) 
perspective continues to have more acceptance in the work of international development 
agencies and governments. 

Our map does not intend to present the approaches as fixed positions. In 
evaluation practice, combinations among them and adaptations to different settings and 
problems, as well as the personal interpretation of the evaluator, might well open new 
space for more imaginative and participatory program evaluations. 



The Latin American context and the possible application of new approaches 

A quick look at the practice of evaluation in developing countries -if the term has 
not become too optimistic or naive— shows the preponderance of very centralized 
evaluations that focus mainly on financial factors and quantitative indicators that are used 
as control systems rather than as means to better the functioning of programs, and that 
rarely include the perspective of beneficiaries (Bamberger 1991). The situation of 
program evaluation and social and educational projects in Latin America do not seem to 
escape from this scenario. Robirosa (1986) indicates that in the region “virtually no use is 
made of evaluation activities by decision-making bodies in the management of social 
action programmes and projects” (19). Even more, it is recognized that “the evaluation of 
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social program and projects is an infrequent activity, if not exceptional” (Cohen and 
Franco 1988, 2) 3 , a phenomenon that has included the education sector (Moncada 1982). 
At the same time, scholars have stressed the importance of evaluating social projects in 
Latin America given the traditional lack of resources of the institutions in charge of social 
promotion (Espinoza Vergara 1980), and particularly in a situation of economic regional 
crisis (Cohen and Franco 1988). With regard to educational evaluation in particular, the 
situation in Latin America suggests a very weak relation between research and decision 
making processes, which might be related to the non-existence of systematic evaluation 
mechanisms of educational innovations and the lack of administrative capacity in the use 
of the information produced by educational research (Corvalan 1988). 

The question about the possible application of the aforementioned approaches to 
educational experiences in Latin America can be answered only within the context of 
evaluative practice. However, it is possible to make some comments about how these 
new approaches might respond to some of the problems or challenges that have been 
identified in the Latin American scenario, taken into account that "the way the nations 
adapt evaluation to suit their institutions not only can, but should vary if evaluation is to 
produce findings that are meaningful within a given cultural and political context" 
(Chelimsky 1994, 344). 

We pointed out before that a common characteristic of five of the seven new 
approaches is that they acknowledge and seek to incorporate multiple perspectives. The 
importance of multiple perspectives is related not only to the search for more flexible 
theoretical frameworks, but also with the need of reflecting complex social and cultural 
realities, which cannot be explained from only one perspective. Latin America contains 
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societies characterized by profound cultural and linguistic gaps; societies that include 
post-modern features along with pre-modem elements (Torres and Puiggros 1995). This 
diversity would make advisable, (together with multiple perspectives) an approach like 
the one suggested by Klitgaard (1995) -the analogy of the soil scientist- that would 
attend especially to the socio-cultural context, without attempting to fully contain it but to 
consider how it influences the program or decisions under consideration. Social 
cartography, on the other hand, would help to make possible a better understanding of the 
scenarios in which diverse “constructions” or visions of reality take place, as, for 
example, those of central governments and those of local actors. 

The rich cultural diversity of Latin America also relates to the need of global and 
decentralized evaluations. It has been indicated that in the region the evaluations are of 
minor utility because of the “absence of any analysis which takes into account the 
historical background of the beneficiary group, that is to say, which identifies and 
classifies the main relationships existing between the phenomenon concerned and the 
society in which it originates... the “omission” of an overall frame of reference (social 
reality) for both the project itself and the actual evaluation process” (Montejo et al. 1986, 
80). In a similar way, Aguerrondo (1993) points to the need of evaluating educational 
quality in a global manner and considering that the meaning of quality might vary 
according to different social contexts. The author emphasizes that the evaluation of 
quality can not be limited to the measurement of results, but should also include 
qualitative information within an interpretative framework that takes into account the 
different dimensions of educational quality. These statements seem to be, again, a call for 
perspectivist approaches in evaluation, approaches that provide a global interpretative 
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framework. Social cartography as well as the organizational perspectives method appear 
as appropriate tools for this purpose. Meanwhile, the participatory character of other 
approaches (concept mapping, participatory self-evaluation, organizational learning, soil 
scientist analogy) would help to recognize local conditions, and might help to make 
evaluations more meaningful and supportive of social and educational change. 

On the other hand, participatory evaluations, as Espinoza Vergara (1980) points 
out, should be “the logical step of an integral action that considers the participation of the 
people from the origins of the very same action” (27), rather than seeking the 
participation of the beneficiaries in just one aspect of the program. In their study of the 
effects of the application of a participatory approach geared toward the empowerment of 
indigenous Mexican communities, Brunner and Guzman (1989) point out that such an 
approach can be successfully applied only when the emancipation of the participant 
groups is sought and when these groups are already prepared to assume the responsibility 
of the administration of the project. Vargas Vargas (1991) argues in favor of participatory 
evaluations in programs of social development in the Central American context. In a 
characterization that could be applied to Latin America in general, this author points to 
the structural conditions of poverty, oppression, hunger, and discrimination. Given those 
conditions, “evaluation methodology must help to develop more critical ability, 
empowerment and self-determination at the grassroots levels”. (Vargas Vargas 1991, 
269). The method of participation proposed by concept mapping might fit better to 
situations in which only a restricted participation is allowed, since it can be used as a 
technique to measure the opinions or perceptions about a program or project, even though 
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we doubt its effectiveness for an evaluation to incorporate major collaboration by ground- 
level agents and beneficiaries. 

It has also been pointed out that evaluations need to take into account the political 
dynamic that, particularly in Latin America, prevails over the administrative dynamic, 
and sets conditions for the way in which social programs are constructed in practice 
(Picado 1989). In this sense, the intertextual evaluation proposed by Roe (1992) appears 
as a method appropriate to apply in cases of programs or policies that generate a high 
degree of conflict, for example sex education in the public schools, or the entrance 
requirements for public universities, to name but two. 

The organizational learning approach does not seem to exist in the educational 
structures of Latin America, but there is no reason to dismiss possible innovations in this 
direction. Reimers (1991) argues that the lack of administrative continuity conspires 
against institutional learning, and that one of the clues to improve educational 
management in Latin America is “to create the conditions in order to make the ministries 
of education learn from their own experience” (p. 111). Evaluations linked to the 
organizational learning concept could be highly productive in those systems in which 
initiatives of school autonomy have been carried out or are being attempted. The 
participatory self-evaluation method might be a useful tool to promote the interchange of 
ideas among educators, parents, students, and administrators in the context of school 
projects that seek to include local actors. On the other hand, schools in Latin America, 
with their variety of administrative models, could be appropriate settings to apply a 
combination of the evaluation based on organizational perspectives and the knowledge of 
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the different actors (teachers, students, parents, administrators), taking into account the 
local context and the expectations and conditions set at the central political instances. 

Social cartography and intertextual evaluation are also approaches that could 
contribute to a critical patterning of the educational discourses that prevail in the region. 
In this sense, Torres and Puiggros (1995) exhort Latin American educators to question 
their assumptions and concepts through the deconstruction of their discourses, as a 
inevitable step to come out from the current “organic crisis” of education in the region 4 . 



Conclusions 

The new approaches presented here generally favor evaluations that take into 
account and reflect different perspectives, and at the same time allow a wide participation 
of the actors involved in the program. Moreover, they are open to be combined with other 
approaches. In this way, they provide a flexible set of tools for evaluators and other 
actors involved in educational and social projects. 

When the conditions of autonomy and meaningful participation are present, 
schools in Latin America would be appropriate scenarios for evaluations linked to the 
concept of organizational learning and for participatory self-evaluation. The method of 
concept mapping appears as a useful tool to measure the perceptions of ground-level 
agents and/or beneficiaries of a program and to graphically represent them As such, it 
could also help to promote participation, reflection, and discussion of ideas within 
educational institutions. Projects of educational development carried out by international 



agencies or central governments could benefit from the “soil scientist” approach by 
integrating the knowledge of experts and local beneficiaries. The methods of social 
cartography and intertextual evaluation may be too complex to be applied in a 
participatory way; however, their use could contribute to the analysis of educational 
discourses and to better comprehend scenarios of contradictory visions or conflicting 
scenarios about a program or a given policy. The approach of organizational perspectives, 
in turn, could facilitate the evaluation of programs in which bureaucratic structures, as 
ministries of education, have a lot of influence. 

It seems highly useful to consider the application of new evaluation approaches 
given the social and educational conditions in Latin America. The seven approaches we 
have analyzed offer in general the prospect of more participatory and constructivist 
evaluations by integrating the knowledge and perspectives of different actors. In contrast 
to the traditional authoritarian approaches that have prevailed in Latin America, 
approaches that accept multiple perspectives and that promote more local people’s 
control of projects might allow for a questioning of the objectives and logic of a given 
program and for a search for alternatives. Evaluators today need to be aware of the 
theoretical and political implications of multiple new approaches, and of the possibilities 
of combinations among them and with other, more traditional, methods. 
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Notes 



We thank Martha Mantilla for helping in the edition of this paper, and Andrea Vidal for 
computer assistance in the elaboration of Figure 10. 

1. Roe bases his proposal on several works of Riffaterre. For an understanding of the 
main concepts of Riffaterre ’s theory see "Fear of theory," New Literary History , vol. 21, 
no.4 (1990): 921-938, and Fictional Truth, Baltimore, MD: The John Hopkins University 
Press, 1990. 

2. The works of Roe and Klitgaard do not include figures. Klitgaard uses formulas in his 
argumentation in favor of the analogy of the soil scientist. In the case of Roe, his 
approach assumes that reality is built through the text, not through graphics or maps. 

3. The quotations of texts of Spanish origin are the translation of the first author. 

4. Torres and Puiggros, unfortunately, do not deconstruct their own text by way of 
application and illustration. 
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